Skip to content

perf(table): compute partition residuals for local scan tasks - #1971

Merged
zeroshade merged 4 commits into
apache:mainfrom
fallintoplace:perf/local-scan-partition-residuals
Sep 1, 2026
Merged

perf(table): compute partition residuals for local scan tasks#1971
zeroshade merged 4 commits into
apache:mainfrom
fallintoplace:perf/local-scan-partition-residuals

Conversation

@fallintoplace

@fallintoplace fallintoplace commented Aug 29, 2026

Copy link
Copy Markdown
Contributor

What

  • Compute partition residuals during local planning.
  • Cache one evaluator per partition spec and reuse it for every data file in that spec.
  • Use strict and inclusive projections to return AlwaysTrue, AlwaysFalse, or the remaining row filter for each task.
  • Cover identity, time, truncate, bucket, null, set, boolean, and transformed filter predicates. Unsupported transforms stay conservative.
  • Read only the fields needed by each task residual, so a fully satisfied partition filter does not add filter-only columns to the file read.
  • Include the identity-filter coverage from perf(table): elide identity partition filters #1966: nested fields, decoded decimal partition values, mismatched transforms, and end-to-end reads.

Example

For ts >= ... AND ts < ... on day(ts) partitions:

  • Interior partitions get AlwaysTrue.
  • Boundary partitions keep only the needed timestamp predicate.
  • Partitions outside the range get AlwaysFalse.

Benchmark

Apple M1 Pro, Go 1.26.3, GOMAXPROCS=4. Medians of six 1-second runs on 2c72b57.

Arrow reads: 32,768 rows from in-memory Parquet, selecting only payload.

Filter Task residual ms/op MB/op allocs/op
Identity only Original filter 2.367 8.95 944
Identity only AlwaysTrue 1.479 5.55 521
Mixed Original filter 3.388 19.76 1,830
Mixed Amount predicate only 2.941 17.33 1,486
  • Identity only: 37.5% less read time and 37.9% fewer allocated bytes.
  • Mixed filter: 13.2% less read time and 12.3% fewer allocated bytes.
  • These compare the Arrow read step with and without task residuals. Planning is measured separately.

Residual evaluation for 4,096 files, reusing one evaluator:

  • Identity: 0.714 ms/op, 393,216 B/op, 18,432 allocs/op.
  • Day range: 1.153 ms/op, 553,801 B/op, 31,950 allocs/op.
GOMAXPROCS=4 go test ./table -run '^$' -bench '^(BenchmarkPartitionResidualPlanning|BenchmarkArrowScanTaskResidual)$' -benchmem -benchtime=1s -count=6

Checks

  • go test ./table/... -count=1
  • go test -race ./table -count=1
  • go test ./... -run=^$ -count=1
  • go vet ./...
  • go run github.com/golangci/golangci-lint/v2/cmd/golangci-lint@v2.12.2 run --timeout=10m

🔥

Signed-off-by: Minh Vu <vuhoangminh97@gmail.com>
Signed-off-by: Minh Vu <vuhoangminh97@gmail.com>
Signed-off-by: Minh Vu <vuhoangminh97@gmail.com>
@fallintoplace
fallintoplace force-pushed the perf/local-scan-partition-residuals branch from 635798d to a8d1268 Compare August 30, 2026 22:18
Signed-off-by: Minh Vu <vuhoangminh97@gmail.com>

@zeroshade zeroshade left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The residual path is conservatively wired through planning, task-specific column projection, and Arrow row filtering. Approving, with one test-matrix follow-up.

Residual computation is the kind of change where being slightly wrong in either direction is silent: too weak returns rows that should have been filtered, too strong drops matching rows. So I focused on the conservatism of the projection rules:

  • Identity elision is safe — the partition value fully determines the predicate, so dropping it is correct.
  • Lossy transforms stay conservative. Truncate and bucket retain the original predicate rather than eliding it, which is the required behaviour since the transform doesn't uniquely determine the source value.
  • Boolean composition — AND/OR/NOT simplification behaves correctly over the mixed cases.
  • Nulls and NaN are preserved rather than silently folded, including null partitions, null predicates, NaN partitions and NaN literals.
  • Missing partition values and unsupported cases leave the expression unchanged, which is the right default.
  • Column projection only narrows to fields the task residual actually needs, while retaining original-filter fields for nil-residual tasks — so a task that didn't get a residual still filters on everything it did before.
  • An end-to-end identity residual read covers the whole path rather than just the expression algebra.

Minor — extend the transform matrix (table/partition_residual_test.go:226)

The matrix covers identity, truncate, bucket, and day/time boundaries, but not year, month, or hour with matching, interior, and non-matching partition values. These delegate through the same shared time-transform projection path, so I don't suspect a defect — but a boundary regression in one of them would silently change which rows get scanned, and that's precisely the failure mode residuals can hide. Please add the same boundary and equivalence cases for year/month/hour (including timestamp and timestamp-ns inputs where supported), ideally with an end-to-end read for one of them. Fine as a follow-up.

Benchmarks are committed for both the scanner and residual paths, with body numbers on M1 Pro / Go 1.26.3 / GOMAXPROCS=4 covering identity-only and mixed-filter comparisons.

The one failing check is the known pre-existing SQLite flake #1793 in catalog/sql, which this diff doesn't touch — unrelated, and nothing for you to do. All 4 commits signed off.


This review was drafted by an AI-assisted tool and confirmed by an Apache Iceberg Go maintainer, who has read the findings and signed off. If something feels off, please reply on the PR and a maintainer will follow up.

More on how to contribute to Apache Iceberg Go: CONTRIBUTING.md

@zeroshade
zeroshade merged commit d4f3619 into apache:main Sep 1, 2026
14 of 15 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants